Level: Tehnical
Abstract:
This presentation explores a multi-year research project using Large Language Models (LLMs) to uncover hidden threats in the open-source software supply chain. What began in 2024 as an experiment in automating changelog analysis evolved into one of the most effective techniques we’ve seen for discovering silent vulnerabilities and active malware. Our research even allowed us to observe North Korean APT group Lazarus as they deployed malware in a supply chain attack.
Key findings include:
- Discovery of 1,500+ security vulnerabilities in popular open-source packages
- None had CVEs or public disclosure
- 25% rated high or critical severity
- Included widely used libraries like Axios and Apache eCharts
- Exposure of “silent patching”
- Maintainers fix security issues without public notification
- Users remain vulnerable without realizing they need to update
In parallel, we used LLMs to analyze newly published packages on public registries like NPM, PyPi and the VSCode Marketplace by detecting:
- Suspicious descriptions and metadata
- Unexpected obfuscation
- Unusual dependency patterns
- Behavioral signals combined with traditional scanning
This approach uncovered thousands of malicious packages uploaded monthly, including activity linked to state-sponsored APTs. It also helped uncover multiple high-profile supply chain attacks in 2025, including:
- The compromise of debug and chalk on NPM, delivering malware through packages totaling ~2 billion weekly downloads
- Shai-Hulud, a self-propagating worm on NPM
- The compromise of the official XRP cryptocurrency SDK on NPM
- And many more
The talk provides a technical deep dive into how the LLM-based detection systems were designed, the validation and triage workflows used to reduce false positives, the most impactful discoveries from the research, and what these findings mean for vulnerability disclosure, software supply chain defense, and the growing role of LLMs in real-world threat hunting.
Bio:
Mackenzie Jackson aka Mackenzie is a developer advocate with a passion for DevOps and code security. As the co-founder and former CTO of a health tech startup, he learnt first-hand how critical it is to build secure applications with robust developer operations.
Today as the Developer Advocate at GitGuardian, Mackenzie is able to share his passion for code security with developers and works closely with research teams to show how malicious actors discover and exploit vulnerabilities in code.